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Introduction 


Training  specific  surgical  skills  on  simulators  has  been  proven  to  bring  a  better-prepared 
student  to  human  operating  room,  and  when  the  simulator-trained  student  performs  a  portion  of  a 
procedure  fewer  errors  are  made  when  compared  to  a  learner  who  has  not  been  trained  on  a 
simulator.  This  current  study  seeks  to  further  this  work  by  first  developing  a  curriculum  for  traininc  an 
entire  procedure,  laparoscopic  cholecystectomy,  using  simulation  technologies  and  integrating 
cognitive,  psychomotor  aspects  of  full  procedure  training,  and  second,  to  test  the  effectiveness  of 
curriculum-based  training  through  a  multicenter,  international  research  group,  the  MASTER  grouf . 

Body 

Thus  far  the  laparoscopic  cholecystectomy  curriculum  including  the  cognitive  and  psychon  otor 
components  has  been  developed  and  validated  locally  (See  Appendix  A).  Expert  performance  ie  rels 
have  been  established  through  three  separate  surveys  of  advanced  laparoscopic  surgeons.  The  first 
survey  was  conducted  during  the  2003  Annual  Meeting  of  the  Society  of  Laparoscopic  Surgeons 
(SLS)  where  100  surgeons  who  had  performed  in  excess  of  100  advanced  laparoscopic  procedu  es 
were  tested  on  the  MIST-VR.  The  second  occurred  during  the  2003  Clinical  Congress  and  used  he 
same  methodology,  and  the  third  during  the  2004  Annual  SAGES  meeting.  Results  from  these 
surveys  have  been  published  and  these  data  are  now  part  of  a  database  for  use  in  setting 
benchmarks  for  performance  on  the  MlST-VR  (Appendix  B  &  C). 

The  methodology  for  executing  the  study  at  multiple  sites  has  been  developed  and  distribi  ited 
(Appendix  D).  This  “cookbook"  serves  as  a  step-by-step  guide  for  setting  the  MIST-VR  and 
establishing  local  expert  levels  for  collaborating  institutions. 

Local  IRB  and  DOD  HSRRB  approval  has  been  acquired  at  the  lead  center  and  other  site  ;  are 
in  the  process  of  submitting  their  local  IRBs. 

Results  from  the  lead  center  have  shown  overall  reduction  in  both  time  and  errors  with 
curriculum-  trained  group. 
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Several  problems  have  slowed  progress  with  this  project: 


1 .  Technical  challenges  have  become  apparent  during  this  time  and  issues  with  the 
durability  of  simulators  has  emerged.  The  MIST  is  not  well  supported  by  industry  ar  d 
there  is  a  need  to  change  the  simulation  platform  to  a  better  and  more  durable 
simulator. 

2.  Programmatic  challenges  have  also  hindered  this  project.  Time  required  to  complel  e 
DOD  HSRRB  application  and  to  receive  approval  from  same  far  exceeded  original 
expectations.  Institutional  IRB  approval  has  been  problematic  due  to  confusion 
regarding  education  research.  In  most  cases,  education  research  is  except  from  fui 
IRB  review.  In  this  project,  the  research  subjects  are  students,  not  patients.  When  he 
student  is  in  the  operating  room  and  their  performance  is  being  videotaped,  the 
attending  surgeon  is  always  present  and  supervises  the  student  at  all  times.  If  the 
student’s  performance  of  the  operation  falls  below  what  would  be  acceptable  for  the 
patient,  the  attending  surgeon  takes  over  the  conduct  of  the  operation.  This  is  the 
standard  of  care  today  in  all  centers  with  surgical  trainees.  This  take  over  by  the 
attending  is  actually  a  data  point  which  is  recorded  when  the  videotaped  procedure  is 
scored  by  blinded  reviewers.  IRB  committees  have  found  this  methodology  confusi  ig 
thinking  that  patients  were  being  exposed  to  untrained  surgeons  without  oversight. 
Considerable  time  was  required  to  educated  IRBs  on  this  experimental  design. 

3.  Turnover  with  collaborating  surgeons  moving  from  one  academic  center  to  another  has 
meant  that  some  of  the  original  collaborating  centers  no  longer  have  the  skill  and 
leadership  to  participate  in  this  research.  This  meant  that  new  centers  had  to  be 
recruited. 
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Key  Research  Accomplishments 


1.  Designed  curriculum  including  cognitive  and  psychomotor  components 

2.  Validated  curriculum  and  presented  resuits  at  national  meeting 

3.  Established  expert  performance  levels  and  published  /  presented  results 

4.  Secured  IRB  and  HSRRB  approval  at  lead  center 

5.  Distributed  execution  methodology  to  collaborating  sites 

6.  ACS  presentation  on  preliminary,  single  site  results 

7.  Methodology  being  used  by  ABS  and  others  in  curriculum  design 

8.  Results  from  lead  center 
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Reportable  Outcomes 


Manuscripts 

1 .  Gallagher,  A.  G.,  C.  D.  Smith,  et  al.  (2003).  "Psychomotor  skills  assessment  in  practicing  surge  )ns 
experienced  in  performing  advanced  laparoscopic  procedures."  Journal  of  the  American  Collegg  ^ 
Surgeons  197(3):  479-88. 

2.  Gallagher,  A.  G.,  A.  B.  Lederman,  et  ai.  (2004).  "Discriminative  validity  of  the  Minimally  invasivf 
Surgical  Trainer  in  Virtual  Reality  (MIST-VR)  using  criteria  levels  based  on  expert  performance.' 
Surgical  Endoscopy  18(4):  660-5. 

Abstracts 

1.  McClusky,  D.  M.,  C.  D.  Smith,  etal.  (2004).  Virtual  Reality  Training  Improves  Operating  Room 
Performance  of  PGY  1  &  2  Surgical  Residents:  Results  of  a  Prospective,  Randomized,  Double- 
Blinded  Study  of  the  Complete  Laparoscopic  Cholecystectomy.  Proceedings  of  the  Surgical 
Forum,  2004. 

2.  Van  Sickle,  K.  R.,  C.  D.  Smith,  et  al.  (2005).  Setting  National  Benchmark  Proficiency  Levels  foi 
Laparoscopic  Performance  Using  Simulation:  The  Results  from  the  2004  SAGES  MIST-VR 
Learning  Center  Study.  SAGES  2005  Annual  Meeting.  Accepted  for  Oral  Presentation. 

Presentations 

1 .  Virtual  Reality  Training  Improves  Operating  Room  Performance  of  PGY  1  &  2  Surgical  Reside  its: 
Results  of  a  Prospective,  Randomized,  Double-Blinded  Study  of  the  Complete  Laparoscopic 
Cholecystectomy.  American  College  of  Surgeons  Surgical  Forum,  New  Orleans,  LA,  October  3, 
2004. 
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Conclusions 


The  full  procedure  curriculum  has  been  developed  and  validated.  Lead  center  local  IRB 
approval  has  been  secured.  New  collaborating  centers  are  being  recruited.  It  is  anticipated  that  tl  le 
24  study  subjects  will  be  enrolled  and  studies  completed  in  the  next  6-9  months.  Completion  of  th  s 
work  will  have  significant  impact  in  several  ways: 

1 .  This  will  be  the  first  study  to  validate  the  benefit  and  role  of  simulation  for  full  procedure 
training, 

2.  With  the  focus  on  curriculum,  and  not  just  a  specific  surgical  skill,  this  work  will  catalyze 
simulation  developers  to  progress  past  developing  technologies  that  simply  train  a  psychomotor  s  dll, 
but  rather,  offerings  that  incorporate  cognitive  and  psychomotor  skills  within  a  curriculum  that  whe  n 
integrated  into  the  simulators  will  provide  a  package  more  appealing  to  the  surgical  and  procedur  il 
educators. 

3.  With  the  significant  logistic  issues  of  conducting  multicenter  educational  research  resol’  ed 
through  this  project,  this  methodology  can  be  easily  reproduced  thereby  providing  a  readily  avaiic  ble 
mechanism  to  generate  significant  numbers  of  subjects  in  short  periods  of  time  to  further  validate 
simulation  strategies  for  industry  and  educators  alike. 

References 

1 .  Gallagher,  A.  G.,  C.  D.  Smith,  et  al.  (2003).  "Psychomotor  skills  assessment  in  practicing 
surgeons  experienced  in  performing  advanced  laparoscopic  procedures."  Journal  of  the 
American  College  of  Surgeons  197(3):  479-88. 

2.  Gallagher,  A.  G.,  A.  B.  Lederman,  et  al.  (2004).  "Discriminative  validity  of 
the  Minimally  Invasive  Surgical  Trainer  in  Virtual  Reality  (MIST-VR)  using 
criteria  levels  based  on  expert  performance."  Surgical  Endoscopy  18(4):  660-5. 
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APPENDIX  A 


Presented  at  2004  ACS  Surgical  Forum 

Virtual  Reality  Training  Improves  Operating  Room  Performance  of  PGY  1  & 
2  Surgical  Residents:  Results  of  a  Prospective,  Randomized,  Doubie- 
Biinded  Study  of  the  Complete  Laparoscopic  Choiecystectomy. 

Objective 

To  demonstrate  that  virtual  reality  (VR)  training  improves  tedinical  skills  of  junior 
residents  in  the  operating  room  during  completion  of  the  full  laparoscopic 
cholecystectomy  (LC). 


Summary  Bacl^round  Data 

VR  training  has  been  demonstrated  to  improve  technical  skills  of  PGY  1-4  residents 
during  the  final  dissection  portion  of  a  LC. 


Methods 

Eleven  surgical  residents  (PGY  1  &  2)  had  baseline  psychomotor,  visio-spatial  and 
perceptual  abilities  assessed  and  were  dien  randomized  to  either  Control  (n=6)  or  VR 
training  with  MIST  VR  simulator  (n=5)  until  performance  criterion  levels  established  by 
experienced  laparoscopists  were  achieved.  All  subjects  performed  a  video-recorded  LC 
with  a  supervising  attending  surgeon  blinded  to  training  status.  Video-recordings  were 
assessed  by  two  surgeon  investigators  blinded  to  subject  identity  and  training  status  and 
scored  using  pre-defmed  errors  for  exposure  (n  =  11),  clipping  and  dissection  (n  =  12), 
and  dissection  of  the  gallbladder  from  the  liver-bed  (n  =  8)  with  inter-rater  reliability 
(IRR)  >0,8  (mean  IRR  =  0.96), 


Results 

VR  trained  subjects  completed  the  full  LC  20%  faster  than  controls  (31.2  v.  39.2 
minutes)  and  made  half  as  many  errors  during  exposure  of  the  cystic  duct  structures  (5.4 
v.  10,  p  <  0.04)  and  dissection  of  the  gallbladder  off  the  liver-bed  (4  v.  7.2,  p  <  0.03). 
Overall,  controls  made  twice  as  many  intra-operative  errors  (10  v.  5.4,  p  <0.04)  with  four 
times  the  variability  of  VR  trained  subjects. 

Conclusion 

Criterion  based  VR  training  for  junior  surgical  residents  transfers  to  reduced  intra¬ 
operative  errors  and  greater  performance  consistency  for  the  entire  LC. 


APPENDIX  B 


EDUCATION 


Psychomotor  Skills  Assessment  in  Practicing 
Surgeons  Experienced  in  Performing  Advanced 
Laparoscopic  Procedures 

Anthony  G  Gallagher,  PiiD,  C  Daniel  Smith,  MD,  FACS,  Steven  P  Bowers,  MD,  Neal  E  Seymour,  MD,  FACS, 
Adam  Peareon,  BSc,  Steven  McNatc,  MD,  David  Hananel,  BSc,  Richard  M  Satava,  MD,  FACS 


BACKGROUND:  Minimally  invasive  surgery  (MIS)  hai  introduced  a  new  and  unique  set  ofpsychomotor  skills  for 
a  surgeon  to  acquire  and  master.  Although  assessment  technologies  have  been  proposed,  precise 
and  objective  psychomotor  skills  assessment  of  surgeons  performing  laparoscopic  procedures 
has  not  been  detailed. 

STUDY  DESIGN:  Two  hundred  ten  surgeons  attending  the  2001  annual  meeting  of  the  American  College  of 
Surgeons  in  New  Orleans  who  reported  having  completed  more  than  50  laparoscopic  proce¬ 
dures  panicipated.  Subjects  were  required  to  complete  one  box-trainer  laparoscopic  cutting  task 
and  asimilar  virtual  reality  task.  These  tasks  were  specifically  designed  to  test  only  psychomotor 
and  not  cognitive  skills.  Both  tasks  were  completed  twice.  Performance  of  tasks  was  assessed  and 
analyzed.  Demographic  and  laparoscopic  experience  data  were  also  collected. 

RESULTS:  Complete  data  were  available  on  195  surgeons.  In  this  group,  surgeons  performed  the  box- 

trainer  task  better  with  their  dominant  hand  (p  <  0.0001)  and  there  was  a  strong  and  statisti¬ 
cally  significant  correlation  between  trials  (r  —  0.47  —  0.64,  p  <  0.0001).  After  transforming 
raw  data  to  z-scores  (mean  =  0  and  SD  =  1)  it  was  shown  that  between  2%  and  12%  of 
surgeons  performed  more  than  two  standard  deviations  from  the  mean.  Some  sm^ons’  per¬ 
formance  was  20  standard  deviations  from  the  mean.  Minimally  Invasive  Surgical  Trainer 
Virtual  Reality  metrics  demonstrated  high  measurement  consistency  as  assessed  by  coefficient 
alpha  (o  =  0.849). 

CONCLUSIONS:  Objective  assessment  of  laparoscopic  psychomotor  skills  is  now  possible.  Sui^ons  who  had 
performed  more  than  50  laparoscopic  procedures  showed  considerable  variability  in  their 
performance  on  a  simple  laparoscopic  and  virtual  reality  task  Approximately  10%  of  surgeons 
tested  perfonned  the  task  significantly  worse  than  the  groups  average  performance.  Studies  such 
as  this  may  form  the  methodology  for  establishing  criteria  levels  and  performance  objectives  in 
objective  assessment  of  the  technical  skills  component  of  determining  surgical  competence. 
(J  Am  Coll  Surg  2003;197:479-488.  ©  2003  by  the  American  College  of  Surgeons) 


The  introduction  of  laparoscopic  or  minimally  invasive 
surgery  (MIS)  dianged  the  way  patients  wanted  their 

This  study  was  supported  with  grants  from  the  American  College  of  Sur¬ 
geons,  Sodety  of  Amencan  CastroiniestinaJ  Endoscopic  Surgeons,  Society  of 
Ldpjuo&copic  Surgeon^  Telemedicine  &nd  Advanced  Technolo^es  Research 
Center  and  Emory  Endosurgery  Unit,  Emory  University  School  of  Medicine. 
Mr.  Hananel  is  an  employee  of  METI,  Inc,  a  company  that  manufactures 
surgical  simulatois,  though  no  METI  products  are  included  in  this  study. 

From  the  Department  Surgery,  University  of  ^K^KIngton  Medical  Center, 
Seatde,  WA  (Gallagher  Pearson,  Satava);  Endoscopic  Research  laboratory, 
BelfWt,  Nonhern  Ireland  (Gallagher);  Emory  Endosurgery  Unit,  Emory  Uni¬ 
versity  School  of  i'dedicine,  Atlanta,  GA  (Smith,  Bowers,  Seymour,  McHart); 
and  Medical  Education  Technologies,  Inc,  Sarasota,  FL  (Hananel). 
Correspondence  address  Richard  M  Satava,  MD  FACS,  University  of  Wash¬ 
ington  Medical  Center,  Department  of  Surgery  Km  #430,  1959  NE  Pacific 
St,  Seattle,  WA  98195. 


surgery  performed  and  the  way  surgeons  thought  about 
doing  surgery.  Although  MIS  conferred  considerable  ad¬ 
vantages  on  the  patient,  it  imposed  major  disadvantages 
on  the  surgeon,  MIS  is  more  demanding,  requiring 
greater  concentration  chan  open  surgery.  Operating 
times  are  longer,  with  increased  surgical  fatigue  and 
stress  from  the  remote  interface  required  with  the  lapa¬ 
roscopic  approach.'  The  surgeon  must  reconstruct  a 
three-dimensional  mental  image  of  internal  organs  and 
structure  from  a  two-dimensional  image  provided  by  the 
charged-couple  device  (CCD)  camera  and  monitor.  Al¬ 
though  the  images  captured  by  the  CCD  device  and 
displayed  on  the  monitor  are  of  very  high  quality,  the 
images  they  produce  are  orders  of  magnitude  poorer 
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Mbrevtations  and  Acronyms 

C!  =  correct  indsion 

la  =  inconea  incision 

MIS  =  minimally  invasive  surgery 

M]STVR=  Minimaltyinvaave  Surreal  Trainer  Virtual 
Reality 

VR  =  virtual  reality 


than  would  be  produced  by  the  eye  under  natural  view¬ 
ing  conditions.^  The  surgeon  also  has  to  cope  with  the 
reduced  degrees  of  freedom  of  the  surreal  instruments 
in  comparison  to  open  surgery.  This  makes  tasks  that  are 
relatively  straightforward  in  open  surgery  (eg,  suturing) 
very  difficult  with  the  minimally  invasive  approach.  Fi¬ 
nally,  the  surgeon  has  to  overcome  or  “automate”  to  the 
counterintuitive  movement  of  instruments  because  of 
the  fulcrum  effea  of  the  body  wall  on  instrument  han¬ 
dling.*  This  means  riiat  when  the  surgeon  moves  his 
hand  to  the  right,  the  working  end  of  the  instrument 
within  view  on  the  monitor  moves  to  the  left  and  vise 
versa.  The  fulcrum  effect  causes  a  fundamental  visual- 
proprioceptive  conflict  that  can  only  be  overcome  with 
extended  practice.*  These  problems  mean  that  mini¬ 
mally  invasive  surgeons  must  operate  at  the  very  edge  of 
their  perceptual,  cognitive,  and  psychomotor  abilities. 

■When  minimally  invasive  surgery  was  first  intro¬ 
duced,  a  number  of  studies  reported  higher  complica¬ 
tions  than  open  approaches,  particularly  during  the  early 
part  of  the  minimally  invasive  surgeon’s  career.*’®  Other 
studies  have  reported  that  complications  persist  even 
with  very  experienced  surgeons.'"®  The  precise  reasons 
for  the  complications  experienced  by  senior  surgeons  are 
unclear.®  It  could  be  that  very  experienced  surgeons  are 
referred  more  difficult  cases,  or  there  is  a  MIS  skills 
deficit. 

Surgeons  in  training,  by  their  very  definition,  have  a 
deficit,  but  these  are  often  overcome  with  training 
in  1-  to  3-day  MIS  courses,  increasing  operative  expo¬ 
sure,  and  mentoring.  But  surgeons  in  training  acquire 
these  skills  at  different  rates,  and,  indeed,  some  may  not 
ever  acquire  a  sufficient  level  of  skill  to  perform  safe 
MIS.  Cuschieri’  has  estimated  that  between  5%  and 
10%  fall  into  this  group.  One  of  the  major  problems  in 
attempting  to  establish  whedier  a  surgeon  in  traininghas 
acquired  the  psychomotor  skills  to  perform  MIS  is  the 
current  absence  ofbenchmarks.  Rosser  and  colleagues'® 
have  used  intracorporeal  suturing  as  an  indicator  of  skill 


level.  A  problem  with  the  results  of  this  assessment  of 
intracorporeal  knot  t)ing  is  that  time  was  the  only 
benchmark  metric.  The  inadequacy  with  this  metric  is 
that  although  a  surgeon  could  tie  a  knot  quiddy,  this 
gves  no  indication  of  the  quality  of  the  knot. 

An  alternative  approach  would  be  to  use  virtual  reahty 
(VR)  tasks  to  assess  performance.  The  advantage  of  this 
approach  is  that  precise  metrics  can  be  extracted  from 
performance  on  these  computer-generated  and 
tracked  tasks.  A  number  of  researchers  have  taken  this 
approach.  Several  studies  have  shown  that  a  VR  trainer 
(the  Minimally  Invasive  Surgical  Trainer  or  MIST  VR, 
Menrice  AB)  was  sensitive  enou^  to  distinguish  be¬ 
tween  surgeons  of  different  levels  of  experience  in  the 
psychomotor  skills  of:  speed  of  performance  (comple¬ 
tion  time),  errors  made,  economy  of  instrument  usage 
(path  length),  and  economy  of  diathenny."''*  As  mani¬ 
fest  by  the  designed  tasks  listed  above,  the  MIST  VR 
simulator  is  designed  to  test  only  perceptual  abilities  and 
psychomotor  skills;  there  is  no  component  of  cognitive 
skills  or  dedsion  making  in  the  simulation.  In  a  later 
study,  Gallagher  and  Satava'*  investigated  these  designed 
psychometric  properties  of  MIST  VR,  They  found  that 
the  simulator  had  a  high  test -retest  reliability  and  a  high 
alpha  coeffident  (ie,  a  measure  of  internal  measurement 
consistency),  and  distinguished  between  surgeons  in 
terms  of  learning  curves  and  variability  of  performance 
(construct  validity).  Ail  of  these  studies  concluded  that 
MIST  VR  could  be  a  useful  device  for  assessing  MIS 
performance  in  the  laboratory,  particularly  because  of  its 
performance  metrics  for  psychomotor  skills. 

Following  the  above  validation  (face,  concurrent, 
construct,  and  content  validity)  of  the  MIST  VR  as  a 
system  to  train  and  assess  psychomotor  skills,  the  simu¬ 
lator  was  evaluated  for  predictive  valiiity.  Is  the  system  a 
valid  prediCTor  of  die  performance  of  psychomotor  (not 
cognitive)  skills  in  the  operating  room?  Seymour  and 
colleagues'*  demonstrated  that  residents  trained  on  the 
MIST  VR  simulator  (as  compared  with  a  control  group 
with  no  simulator  training)  made  fewer  errors  and  used 
less  time  for  the  gallbladder  exdsion  porrion  of  a  lapa¬ 
roscopic  cholecystectomy  on  patients. 

But  a  major  obstacle  to  implementing  MIST  VR  as  an 
assessment  device  is  that  the  studies  that  have  been  con¬ 
ducted  have  used  only  small  numbers  of  participants.  It 
is  also  not  clear  how  an  individual’s  performance  on  a 
VR  task  relates  to  performance  of  an  entire  surgical  pro¬ 
cedure  on  a  real-world  task.  More  important,  diere  is  no 
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Study  to  date  (and  it  may  take  decades)  to  prove  that 
training  on  a  simulator  directly  improves  outcomes  (pa¬ 
tient  safety,  decreased  complicadons,  higher  quality  of 
life). 

The  purpose  of  the  study  reported  here  was  to  bench¬ 
mark  MIS  performance  of  psychomotor  skills  of  experi¬ 
enced  ininimaliy  invasive  surgeons  on  a  VR  cask  and  a 
similar  box-trainer  task.  The  box-trainer  task,  a  simple 
laparoscopic  paper-cutting  task,  vfas  chosen  because  it 
was  first  used  to  empirically  demonstrate  the  impact  of 
the  fulcrum  effect  on  MIS  performance  and  has  subse¬ 
quently  been  used  extensively  to  assess  different  MIS 
training  programs.^’'®'’^  It  has  also  been  demonstrated  to 
have  low  variability,  normally  distributed  performance, 
and  is  sensitive  to  learning  and  errors.  This  r^atively 
simple  laparoscopic  task  also  requires  litde  cognidve  ef¬ 
fort  (ie.  remembering  what  to  do),  so  is  a  good  measure 
of  laparoscopic  psychomocor  performance.  For  the  pur¬ 
poses  of  this  study  a  similar  VR  task  was  constructed.  It 
was  predicted  that  experienced  ininimaliy  invasive  sur¬ 
geons  would  demonstrate  variability  in  their  perfor¬ 
mance  on  both  a  virtual  reality  cask  and  a  box-trainer 
task,  although  the  majority  of  surgeons’  performance 
would  fall  within  a  range  plus  or  minus  two  standard 
deviations  from  the  mean.  Another  prediction  was  chat 
performance  on  the  box-trainer  and  VR  task  would  cor¬ 
relate  strongly  enough  to  allow  only  the  VR  task  to  be 
used  in  future  studies. 

METHODS 

Subjects 

Surgeons  attending  the  2001  annual  meeting  of  the 
American  College  of  Surgeons  in  New  Orleans  parti¬ 
cipated.  Two  hundred  ten  surgeons  were  recruited,  and 
15  surgeons  failed  to  complete  any  single  task.  Their 
data  were  excluded  from  subsequent  analysis.  Demo¬ 
graphic  details  of  surgeons  vdio  participated  are  shown 
in  Table  1. 

Apparatus 
MIST  VR 

The  MIST  VR  system  comprises  a  standard  200-MHz 
PC  with  32  Mb  RAM,  linked  to  a  jig  containing  two 
laparoscopic  instruments  held  in  position-sensing 
gimbles  with  5  d^rees  of  freedom.  This  provides  real¬ 
time  translation  of  the  instrument  movements  to  the 
graphic  display  on  a  15-inch  color  monitor.  An  accu¬ 
rately  scaled  operating  volume  of  10  cm^  is  represented 


Table  1.  Demographic  Details  of  Surgeons  Tested. 


Age  (y) 

30-39 

24.6% 

40-49 

40% 

30-59 

23% 

60-69 

12.4% 

>70 

0% 

Mean  number  of  career 

967  (SD  =  994;  range  20-4,000) 

laparoscopic  cases 

Mean  number  ot  laparoscopic 

144  (SD=  105;range2-400) 

ases  peryear 

Right  hand  dominant 

92.3% 

Male 

86% 

by  a  three-dimensional  cube  on  the  computer  screen. 
The  image  zoom  and  size  of  the  target  objects  can  be 
varied.  Circular  targets  (at  a  diameter  of  12  mm)  appear 
randomly  within  the  operating  volume  and  can  be 
“grasped"  and  “manipulated.”  The  MIST  VR  trainer 
recorded  time,  error,  and  economy  of  instrument  move¬ 
ment  relative  to  the  target  object.  Anovel  MIST  VR  task 
was  developed  for  this  study  and  is  shown  m  Figure  1 .  It 
consisted  of  a  rectangular  virtual  card  with  five  spheres 
along  the  long  edge  and  a  larger  sphere  in  the  middle  of 
the  short  edge.  The  large  sphere  indicated  which  hand  or 
instrument  was  to  be  used  to  grasp  the  card  and  where 
subjects  should  start  excising  the  spheres.  The  hand  or 
instrument  used  to  grasp  the  card  alternated  v.ith  each 
card  completed. 


Figure  1.  The  new  MIST  VR  task  for  the  2001  American  College  of 
Surgeons  objective  assessment  of  laparoscopic  psychomotor  skill 
program.  MIST  VR.  Minimally  Invasive  Surgical  Trainer  Virtual  Real¬ 
ity. 
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Figure  2.  The  portable  laparoscopic  box  trainer. 


Box-trainer  task 

A  standard  laparoscopic  cutdng  task  was  used  as  previ¬ 
ously  reported. Briefly,  a  cut  is  made  in  26  spaces 
clearly  demarcated  by  black  lines  along  the  long  edge  of 
a  sheet  of  paper.  The  task  was  performed  in  a  Portable 
Laparoscopic  Trainer  (3-0  Technical  Services,  Franklin, 
OH)  (Fig.  2).  Ethicon  nontoothed  laparoscopic  forceps 
(Ethicon  Endosurgery,  Cincinnati,  OH)  and  laparo¬ 
scopic  curved  scissors  were  used  to  hold  and  cut  the 
paper.  This  task  has  been  shown  to  be  sensitive  to  differ¬ 
ences  between  experienced  and  junior  surgeons,  is  sen¬ 
sitive  to  learning,  measures  correct  responses  and  errors, 
has  low  variabihcy,  and  has  been  demonstrated  to  con¬ 
form  to  the  Gaussian  distribution. 

Procedure 

Subjects  reported  to  the  testing  booth  in  the  scientific 
exhibition  area  of  the  conference  and  completed  a  ques¬ 
tionnaire  detailing  their  experience  and  training.  After 
confirming  that  they  had  completed  more  than  50  lapa¬ 


roscopic  procedures,  they  were  given  a  unique  identifier 
number  and  then  watched  a  video  recording  explaining 
how  to  perform  the  laparoscopic  tasks,  including  what 
constituted  an  error.  Subjects  rotated  through  four  test¬ 
ing  stations  in  alternate  order,  two  virtual  reality  stations 
and  two  box-trainer  cutting  stations.  The  virtual  reality 
task  required  subjects  to  hold  the  virtual  card  with  one 
instrument  and  excise  the  spheres  by  grasping  them  in 
the  center  vrirh  the  other  instrument.  When  this  was 
done  cotrecdy  the  sphere  disappeared.  The  virtual  real¬ 
ity  task  always  started  with  the  subjects  holding  the 
sphere  with  their  nondominant  hand  and  exdsing  the 
spheres  with  their  dominant  hand.  After  all  five  spheres 
had  been  excised,  a  new  card  appeared  and  was  to  be 
grasped  with  the  dominant  hand.  All  subjects  were  re¬ 
quired  to  complete  four  cards  with  a  total  of  20  spheres. 
Tmiing  was  stopped  between  all  the  spheres  being  ex¬ 
cised  for  one  card  and  a  new  card  appearing.  MIST  VR 
recorded  rime,  errors,  and  economy  of  instrument 
movement. 

The  laparoscopic  cutting  cask  was  placed  horizontally 
in  a  conventional  box  trainer  under  standardized  lapa¬ 
roscopic  conditions.  Subjects  were  required  to  make  one 
incision  between  26  spaces  clearly  demarcated  by  black 
lines  along  the  long  edge  of  the  sheet  of  paper  (US  let¬ 
ter).  Subjects  grasped  the  paper  with  their  nondominant 
hand  and  cut  with  their  dominant  hand  for  the  first  trial 
and  then  swapped  over  for  the  second  trial.  An  error  was 
judged  to  have  been  made  if  a  subject’s  incision  cut 
across  one  of  the  black  lines  or  couched  it.  If  an  incision 
was  so  close  to  the  line  that  the  experimenter  found  it 
difficult  to  make  a  decision  it  was  judged  an  error. 
Twenty  percent  of  sheets  were  rechecked  by  another  in¬ 
vestigator.  The  inter-rater  reliability  was  98%.  Subjects 
were  asked  to  make  as  many  incisions  as  they  could  in 
two  2-minute  periods.  The  laparoscopic  task  was  re¬ 
moved  from  the  box  trainer  after  each  subject  was  fin¬ 
ished.  Timing  commenced  when  subjects  placed  the  pa¬ 
per  between  the  jaws  of  the  sdssors  for  their  first  ination. 
Subjects  were  stopped  after  60  seconds  had  elapsed. 

RESULTS 

The  means  and  standard  deviations  of  surgeon  perfor¬ 
mance  on  the  box  trainer  and  VR  tasks  are  shown  in 
Table  2.  Differences  between  performance  on  the  two 
trials  were  compared  for  significance  with  ANOVA  for 
repeated  measures  and  these  resulB  are  shown  in  Table  3. 

Results  for  the  box-trainer  task  reveal  that  surgeons 
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Table  2.  Me^  Scores  by  Surgeons  on  the  Box  Trainer  and  Virtual  Reality  Tasks  for  Trials  1  and  2  and  Their  Intercorrelation 

Trial  1  Trial  2 

meat  (SD)  mean  (SO)  r _ « _ fva'ne _ P 


Box  tfiintr  mwi« _ 

Correct  incisions _ 19.1  (5.4) _ \7-7  (6) 

Incorrort  incisions _ 3-03  (7.3) _ 5.14  (13) 

MIST  VR  metrics  _ 

Ti^t  ■  98.9  (54.3)  76.51  (43) 

EH;;;  64.3  {54.4)  53  (38.5T 

Economy  of  movement _ 15.7  (8.5) _ 12.75  (6.2) 


MIST  VR,  Minimally  Invasive  Surgical  Trainer  Virmal  Reality. 

made  significantly  more  correct  incisions  (Cl)  with  their 
dominant  hand  in  trial  1  than  they  did  with  their  non¬ 
dominant  hand  during  trial  2.  Th^  also  made  signifi¬ 
cantly  more  incorrect  incisions  (I Cl)  during  trial  2.  The 
percentage  of  ICI  scores  showed  considerable  variability, 
with  standard  deviations  that  were  twice  as  large  as  the 
mean  scores  for  both  trials. 

Performance  as  a  function  of  hand  dominance  was 
not  assessed  separately  for  MIST  VR  because  the  tasks 
alternated  which  hand  did  most  of  the  work.  Across  all 
three  measures  there  was  a  statistically  significant  im¬ 
provement  between  trials  1  and  2.  There  was  consider¬ 
able  variability  in  all  of  the  VR  scores  on  trial  1  but  this 
had  decreased  by  trial  2.  Pearson’s  Product  Moment 
Correlation  Coefficient  was  used  to  assess  how  strongly 
performance  on  the  first  trial  was  related  to  performance 
on  the  second  trial.  These  are  presented  in  Table  1  with 
F  values  and  probability  levels.  Performance  on  trial  1 
was  strongly  and  sutisrically  signifi candy  related  to  per¬ 
formance  on  trial  2  across  both  the  box  trainer  and 
MIST  VR  measures,  with  correlations  ranging  from  r  = 
0.48  to  0.644. 

For  ease  of  analysis  all  scores  for  each  measure  were 
transformed  to  z-scores  using  the  mean  and  standard 
deviation  of  each  measure  for  each  trial.  The  advantage 
of  this  approach  is  that  the  mean  =  0  with  a  standard 
deviation  =  1.  Any  scores  falling  more  than  two  stan¬ 
dard  deviations  either  way  from  the  mean  differ  statisd- 
cally  significantly  at  the  95%  probability  level  from  the 
sample  mean.  Figure  3A  shows  the  distribution  of  Cl 
z-scores  for  trials  1  and  2  (dominant  hand  on  trial  1  and 
nondominant  hand  on  trial  2).  From  the  graph  it  can  be 
seen  that  some  individuals  are  scoring  four  standard  de¬ 
viations  from  the  mean,  indicating  that  they  are  making 
dramatically  fewer  Cl  than  the  mean  performance: 
2.1%  in  trial  1  and  2.6%  of  subjects  in  trial  2.  A  statis¬ 
tical  problem  created  by  these  outliers  is  chat  they  were 


0,644 _ U91 _ 13M _ 0-0001 

0.472 _ 1.191 _ 546 _ 0-0001 

0487  U83  50  0.0001 

■048  TTsi  939  0.0023 

■o353  1382  797  0-0031 


used  in  the  calculation  of  the  z-scores,  so  will  likely  neg¬ 
atively  bias  the  distribution.  To  eliminate  this  problem, 
new  means  and  standard  deviations  were  calculated  for 
each  of  the  metrics  and  for  each  trial  based  only  on  the 
scores  of  surgeons  who  fell  within  plus  or  minus  two 
standard  deviations.  These  are  presented  in  Table  4. 

AH  subsequent  graphs  show  the  raw  z-scores  on  the 
upper  plate  (A)  and  the  biweighted  z-scores  on  the  lower 
plate  (B).  Even  with  the  biweighted  transformation 
shown  in  Figure  3B,  4.1%  of  surgeons  scored  more  than 
two  standard  deviations  away  from  the  mean. 

Figures  4A  and  4B  show  the  distribution  of  scores  for 
ICIs.  Most  surgeons’  performance  bunched  around  the 
mean,  but  both  distributions  exhibit  a  long  tail.  For  the 
raw  z-scores,  5.64%  of  surgeons  scored  more  than  two 
standard  deviations  away  from  the  mean,  but  for  the 
biweighted  mean  this  was  12.3%.  Indeed,  for  the  bi- 
wdghted  distribution  some  surgeons  scored  more  than 
13  standard  deviations  from  the  mean  on  trial  I  and  up 
to  20  on  trial  2. 

Although  the  MIST  VR  time  scores  show  a  more  even 
distribution  (Figs.  5A  and  5B)  than  the  ICI  scores,  they 
also  exhibit  a  long  tail.  For  the  raw  z-scores,  6.3%  and 
7%  of  surgeons  scored  more  than  two  standard  devia- 


Table  3.  Results  of  ANOVAfor  Repeated  Measure  Between 
the  Difference  in  Performance  on  Trial  1  and  Trial  2 


df 

rvalue 

p  Value 

Box  trainer 

metrics 

Correct  incisions 

1.191 

16.45 

0-0001 

Incorrea  incisions 

1,191 

54.6 

0.0001 

MISTVR  metrics 

Time 

1.183 

36.52 

0.0001 

Errors 

1,181 

54.3 

0-0023 

Economy  of 
movement 

1,182 

30.76 

0,0001 

MISTVR,  MinimaJly  Invasive  Surgical  Trainer  Virtual  Reality. 
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Table  4.  Biwelghted  Mean  Scores  by  Surgeons  on  the  Box 
Trainer  and  Virtual  Reality  Tasks  for  Trials  1  and  2 

Trial  2 
mean  (SD) 


mean  (SD) 


Correa  indsions 


Incorcea  incisions 


MIST  VR  metrics 


Time  89.7  (39.49)  68.22  (27.28) 

Errors _ 55.23  (38.5)  47.72  (28.61) 

Ecjnomy  of  movement  14.38  (6.12)  11.78  (4.3) 

MIST  VR,  Minimally  Invasive  Surgical  Train®  Virtual  Reality. 

deviations  away  from  the  mean  fot  the  raw  z-scotes, 
respeaively.  This  inaeased  to  10%  and  10.7%  for  the 
biweighted  distribution.  Indeed,  some  surgeon  perfor- 
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Figure  3.  (A  and  B)  Distribution  of  z-scores  for  correct  Irrclsions 
made  during  the  two  2^nlnute  trials. 


tions  from  the  mean  for  trials  1  and  2,  respectively.  The 
biweighted  z-distribution  exhibited  11.6%  and  8.6%. 
No  surgeon  performed  more  than  five  standard  devia¬ 
tions  from  the  mean. 

MIST  VR  error  scores  are  presented  in  Figures  6A  and 
6B.  Most  surgeon  performance  bunches  around  the 
mean,  but  both  distributions  exhibit  along  tail.  In  trials 
1  and  2,  5.8%  and  5-4%  scored  more  than  two  standard 
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Figure  4.  (A  and  B)  Distribution  of  z-soores  for  incorrea  Incisions 
made  during  the  two  2-minute  trials. 
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Rgure  5.  (A  and  B)  Distribution  of  2-scores  for  lime  perfbnnance 
metric  on  MIST  VR  for  trids  1  and  2.  MIST  VR,  Minimally  Invasive 
Surgical  Trainer  Virtual  Reality. 
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Figure  6.  (A  and  B)  Distribution  of  z-scores  for  error  performance 
tT>etric  on  MIST  VR  for  trials  1  and  2.  MIST  VR.  Minimally  Invasive 
Surgical  Trainer  Virtue  Reality. 


mance  fell  seven  standard  deviations  from  the  mean  for 
both  trials  1  and  2. 

Economy  of  instrument  movement  exhibited  a  simi¬ 
lar  pattern  to  the  error  scores.  Figures  7A  and  7B  show 
that  most  of  the  scores  bunch  around  the  mean,  and, 
similar  to  the  other  scores,  exhibit  a  long  tail.  The  top 
plate  (Fig.  7A)  shows  that  5.8%  and  5-3%  of  surgeons  in 
trials  1  and  2  performed  more  than  two  standard  devia¬ 
tions  from  the  mean.  For  the  biweighted  distribution  it 
was  11.5%  and  9.6%,  respectively.  One  surgeon  per¬ 
formed  eight  standard  deviations  from  the  mean.  Coef¬ 
ficient  alpha  was  used  to  assess  the  internal  measurement 
consistency  of  the  MIST  VR  metrics  and  was  observed 
to  be  high;  a  =  0.849. 


DISCUSSION 

Most  studies  on  the  objective  assessment  of  laparoscopic 
performance  have  only  used  small  numbers  of  subjects 
(eg,  9,  10,  1 1,  12)  and  when  larger  numbers  have  been 
used  only  crude  measures  time)  have  been  reported. 
This  is  one  of  the  first  studies  to  use  both  a  box-trainer 
task  and  a  virtual  reality  task.  We  also  tested  only  expe¬ 
rienced  laparoscopic  surgeons,  ie,  surgeons  who  claimed 
to  have  performed  more  than  50  laparoscopic  opera¬ 
tions.  Surgeons  performed  better  with  their  dominant 
hand  in  trial  1  of  the  box-trainer  cutdng  task.  On  the 
MIST  VR  tasks  there  was  a  significant  improvement 
between  trials  1  and  2. 

One  goal  of  this  study  was  to  objectively  assess 
whether  some  surgeons  were  performing  worse  than  the 
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Figure  7.  (A  atv3  B)  Distribution  of  z-scores  tor  economy  of  move¬ 
ment  performance  metric  on  MIST  VR  for  trials  1  and  2.  MIST  VR, 
Minlmalty  Invasive  Surgical  Trainer  Vinual  Reality. 


mean  and,  if  so,  how  much  worse.  The  box-trainer  and 
VR  cask  were  chosen  because  they  were  extremely  simple 
and  involved  little  cognition  or  decision  making,  so  they 
should  have  given  a  pure  measure  of  psychometric  per¬ 
formance.  Indeed,  before  the  data  colleaion  started, 
some  of  the  coauthors  expressed  concern  that  the  tasb 
were  too  simple  and  would  not  be  challenging  enough 
for  experienced  laparoscopic  surgeons.  After  data  collec¬ 
tion  started  it  was  evident  that  even  sucii  a  simple  basic 
skill  was  a  sensitive  discriminator.  Some  surgeons  ex¬ 
celled  at  both  tasks,  but  others  had  great  difficulty  per¬ 
forming  either  the  VR  or  the  box-trainer  task.  The  re¬ 
sults  we  have  presented  here  are  from  only  surgeons  who 
completed  some  part  of  the  study.  Fifteen  surgeons  had 


to  be  excluded  because  they  did  not  complete  any  part  of 
the  smdy. 

One  of  the  findings  that  surprised  all  the  authors  was 
how  poorly  some  surgeons  performed  (eg,  up  to  20  stan¬ 
dard  deviations  from  the  mean) .  Reassuringly,  this  num¬ 
ber  was  extremely  small.  But  what  this  study  has  dem¬ 
onstrated  is  that  based  on  objective  metrics  it  is  possible 
to  idendfy  individuals  who  have  laparoscopic  basic  psy- 
diomotor  skill  deficits.  Both  the  box-trainer  task  and  the 
VRtask  achieved  this  and  corroborate  the  veracity  of  the 
results.  But  the  VR  task  generates  the  metrics  automat¬ 
ically  and  records  performance  and  provides  more  com¬ 
prehensive  performance  metrics  in  comparison  to  the 
box-trainer  task,  in  which  the  experimenter  has  to  phys¬ 
ically  count  the  number  of  incisions.  Although  the  inter- 
rater  reliability  observed  in  this  study  was  very  high, 
there  is  always  the  possibikty  of  human  error  with  the 
box  trainer.  The  box-trainer  cask  that  was  used  was  sim¬ 
ple,  with  only  two  possible  outcomes,  both  of  which 
were  dearly  defined.  Had  a  more  complicated  task  been 
used,  the  inter-rater  relidsility  may  not  have  been  as 
high.  MIST  VR  has  also  been  the  subject  of  extensive 
validation  studies*’  '^  and  is  currently  the  best  validated 
VR  system  in  surreal  education.  These  MIST  VR  vali¬ 
dation  studies  used  the  original  six  tasks  that  were  devel¬ 
oped  to  reach  psychomotor  skills  and  instrument  han¬ 
dling  for  laparoscopic  cholecystectomy.  The  study 
reported  here  used  a  novel  task  to  eliminate  the  effect  of 
earlier  exposure.  No  surgeon  had  worked  on  the  new  VR 
task  before  the  American  College  of  Surgeons  2001 
meeting.  One  advantage  that  the  box-trainer  task  had 
over  the  VR  task  was  that  dominant  and  nondominant 
hand  performance  were  assessed  separately.  In  a  new 
study  we  plan  to  assess  dominant  hand  performance  on 
both  the  box  trainer  and  virtual  environments  and  how 
strongly  performance  on  one  correlates  with  the  other. 

Currently  surgjcal  residents  wishing  to  pursue  a  ca¬ 
reer  in  minimally  invasive  surgery  have  no  objective  na¬ 
tional  or  international  benchmarks  at  which  to  aim. 
They  can  only  use  local  attending  surgeon  performance, 
and  frequently  the  local  training  insdtudon  has  not  val¬ 
idated  a  criterion  level  to  achieve,  so  training  is  focused 
on  training  dme,  rather  than  objeedve  criteria  for  profi¬ 
ciency.  What  the  data  from  this  study  have  shown  is  diat 
there  is  considerable  variability  in  surgeon  performance, 
which  we  assume  translates  nadonally.  We  believe  that 
this  issue  -will  become  more  important.  In  the  recent 
study  by  Sejinour  and  Colleagues,'^  the  benefits  of 
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benchmarking  were  clear.  In  a  prospective,  randomized, 
double-blind  trial  of  VR  training  versus  standard  surgi¬ 
cal  training,  the  researchers  found  that  surgical  residents 
trained  on  VR  simulators  made  significantly  fewer  errors 
during  a  laparoscopic  cholecystectomy.  One  of  the  cru¬ 
cial  aspects  of  this  smdy  was  that  the  residents  were 
trained  to  an  objeaively  measured  score  (as  determined 
by  assessment  of  attending  surgeon  psychomotor  perfor¬ 
mance  levels  on  the  VR  simulator),  and  were  required  to 
reach  this  established  performance  level  (aiteria)  on  two 
consecutive  trials  before  being  allowed  to  operate  on  a 
patient The  results  of  that  study  suggested  chat  surr¬ 
eal  residents  clearly  demonstrated  their  ability  to  achieve 
a  criterion  level,  but  some  rook  longer  than  others.  Sur¬ 
gical  residents  should  be  given  clear  guidance  as  to  the 
standard  of  psychomotor  performance  they  should 
achieve  (benchmark)  before  operating  on  a  patient. 

Study  limitations 

One  of  the  limitations  of  this  study  is  that  the  “experts” 
volunteered,  so  they  self-defined  “laparoscopic  sur¬ 
geon.”  We  have  no  objective  independent  information 
on  what  types  of  MIS  they  performed  other  than  their 
self-reports,  and  we  have  no  information  on  their  per¬ 
formance  in  the  operating  room,  ie,  outcomes  data.  An¬ 
other  limitation  of  this  study  was  that  the  casks  were 
probably  too  easy  for  the  expert  laparoscopic  surgeon. 
This  was  most  clearly  demonstrated  on  the  box-trainer 
task,  where  some  surgeons  made  incisions  in  all  26 
spaces,  with  no  errors,  in  less  than  a  minute.  Other  foo¬ 
ters  that  might  have  affected  performance  relate  to  the 
foa  that  itwas  in  a  busy  booth  in  the  convention  exhibit 
hall;  there  were  distractions  from  the  convention  or  they 
may  have  been  tired  after  a  long  flight  or  busy  meeting 
schedule.  Last,  because  of  time  constraints,  demo¬ 
graphic  information  was  self-reported  rather  than  objec¬ 
tively  assessed;  eg,  handedness. 

In  conclusion,  this  study  has  shown  that  using  already 
validated  methods  of  laparoscopic  skills  assessment,  it  is 
possible  to  measure  laparoscopic  psychomotor  perfor¬ 
mance  of  laparoscopic  surgeons  who  had  performed 
more  than  50  laparoscopic  procedures  by  using  a  simple 
box-trainer  task  and  a  virtual  reality  task.  Between  2% 
and  12%  of  surgeons  assessed  fell  more  than  two  stan¬ 
dard  deviations  away  from  the  mean.  The  majority  of 
surgeons  fell  within  the  two  standard  deviations  but  con¬ 
siderable  variability  in  performance  was  observed.  Some 
surgeons’  scores  fell  20  standard  deviations  from  the 


mean.  This  type  of  performance  is  unlikely  to  have  oc¬ 
curred  by  chance  because  a  large  and  statistically  signif¬ 
icant  correlation  was  observed  between  trials. 
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Setting  National  Benchmark  Proficiency  Levels  for  Laparoscopic  Performance  Using  Simulation; 
The  Results  from  the  2004  SAGES  MiST-VR  Learning  Center  Study 

Kent  R.  Van  Sickle,  M.D..  E.  Matt  Ritter.  M.D..  David  A.  McClusky  Hi,  M.D.,  Andrew  Ledermen, 
M.D-,  Mercedeh  Baghai,  M.D.,  Anthony  G.  Gallagher.  PhD.,  C.  Daniel  Smith,  M.D., 

Emory  Endosurgery  Unit,  Emory  University  School  of  Medicine.  Atlanta,  GA 

Background:  The  Minimally  invasive  Surgical  Trainer  Virtual  Reality  (MIST-VR)  (Mentice, 
Gottenberg,  Sweden)  has  been  well  validated  as  a  training  device  for  laparoscopic  skills. 

Training  to  a  levei  of  proficiency  on  the  simulator  has  been  demonstrated  to  significantly  improve 
objectively  assessed  operating  room  performance  during  laparoscopic  cholecystectomy.  The 
purpose  of  this  project  was  to  establish  a  national  standard  of  proficiency  on  the  simulator  based 
on  the  performance  of  experienced  laparoscope  surgeons,  Methods:  Surgeons  attending  the 
SAGES  2004  Annual  Meeting  who  had  performed  more  than  100  laparoscopic  procedures 
volunteered  to  participate  and  were  tested  in  the  SAGES  Learning  Center.  All  subjects 
completed  a  demographic  questionnaire  to  assess  laparoscopic  and/or  MIST-VR  experience. 
Each  subject  performed  two  consecutive  trials  of  the  MIST-VR  Core  Skills  1  program  on  medium 
settings  (six  basic  tasks  of  increasing  difficulty;  acquire  place  (AP),  transfer  place  (TP),  traversal 
(TV)  withdrawal  insert  (Wl),  diathermy  task  (DT).  manipulate  diathermy  (MD)).  Trial  1  was 
considered  a  “warm-up’  and  Trial  2  functioned  as  the  test  trial  proper.  Subject  performance  was 
scored  for  time,  errors  and  economy  of  instrument  movement  for  each  task,  and  a  cumulative 
total  score  was  calculated.  Results;  57  surgeons  participated  in  the  study,  com|:^ete  data  is 
available  for  42.  Trial  2  data  expressed  as  meaniSD;  time  in  seconds;  other  values  unitless. 
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Errors  L 


5.3±2.0 


2.CI±0,3 


27±0.7 


2.4±0.6 


2.8±0.8 


5.1±1.8 


2.0±0.5 


2.4±0.6 


2.3±0.7 


Errors  R 


4.1±3.2 


7.8±6.0 


11.2±10.3 


4,4±5.7 


Tot.  Time 


12.5±2.8 


18.4±5.1 


36.6±14.5 


Score 


26.7±9.2 


38.4±14.0 


73.8±34.9 


27.3±11.4 


41.6±11.4 


160.7±92.9 

374.8±134 


4.7±4.1 


7.7±5.1  r8.2±5.6  20.5±4.7 _ 


51.9±35.3  53,3±34.1  1  50.9±26.9 


MIST-VR  Core  Skills  1  Total  Score  _ . _ 

Conclusion;  National  benchmark  proficiency  levels  for  laparoscopic  skills  have  now  been 
established  by  experienced  laparoscopic  surgeons  using  the  MIST-VR  simulator.  Residency 
programs,  training  centers  and  practicing  surgeons  can  now  use  these  data  to  identify  how  their 
skills  compare  to  laparoscopic  surgeons  nationwide,  and  to  set  performance  goals  accordingly. 
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Introduction 


The  MASTER  curriculum  based  laparoscopic  cholecystectomy  (CBLC)  multicenter  trial 
is  designed  to  assess  whether  residents  trained  using  a  uniquely  designed  curriculum  that 
includes  virtual  reality  laparoscopic  training  on  the  MIST-VR  make  fewer  errors  in  the 
performance  of  laparoscopic  cholecystectomy  than  residents  who  undergo  standard 
training  without  such  a  curriculum  This  design  requires  that  residents  randomized  to  the 
curriculum  training  arm  of  the  study  train  on  the  MIST-VR  to  a  certain  perform^ce  level 
(or  criterion)  prior  to  performing  human  laparoscopic  cholecystectomy.  Currently,  this 
performance  level  is  not  set.  Consequently,  training  cannot  proceed  until  these  levels  are 
established. 

Based  on  previous  models  assessing  Ihe  utility  of  the  MIST-VR  as  atramng  modality, 
this  study  calls  on  experts,  who  have  performed  at  least  100  laparoscopic 
cholecystectomies,  to  set  the  MIST-VR  performance  criterion  that  curriculum  trained 
residents  will  need  to  adiieve.  The  initial  phase  of  this  trial,  therefore,  is  to  analyze  &e 
MIST-VR  performance  of  experts  at  participating  institutions.  This  guide  is  intended  to 
help  your  institution  collect  the  necessary  data  so  this  analysis  can  occur  as  soon  as 
possible.  Specifically  it  outlines  the  steps  needed  to  configure  your  MiST-VR  for  Inal 
participation,  and  describes  the  process  of  saving  and  packaging  the  data  for  statistical 
analysis  at  Emory. 

Before  you  set  up  your  MIST-VR,  there  are  afew  administrati^'e  issues  to  consider: 

1 .  Your  institution  should  assign  one  MIST-VR  admimstrator  who  can 
configure  the  MIST-VR  software  (e.g.  Frameset),  and  can  troubleshoot 
any  software  or  hardware  complications. 

2.  Although  this  guide  is  designed  for  novice  to  intermediate  MIST-VR 
users,  your  administrator  should  have  a  working  knowledge  of  computer 
function,  with  tiie  ability  to  navigate  through  a  graphical  user  interface 
(e.g.  Windows)  without  difficulty. 

3.  Ideally,  your  administrator  should  provide  the  bulk  of  the  MIST-VR 
support  to  those  experts  helping  to  set  paformance  levels,  as  well  as 
oversee  the  training  of  residents  randomized  to  cumculiun  based  training. 

4.  The  administrator  will  need  the  authority  to  ensure  that  testing  and 
training  is  completed  in  a  timely  fashion.  This  is  particularly  important 
when  dealing  with  experts  whose  schedules  often  compete  with  the 
completion  of  the  MIST-VR  tasks  necessary  to  set  expert  performance 

levels.  -  -  1 

5.  In  order  to  maintain  anonymity,  your  experts  will  be  identified  using  tnal 

specific  identifiers  (e.g.  Expert  1,  Expert  2,  Expert  3  etc  ... ). 

We  hope  that  this  guide  will  be  helpful  in  your  endeavors.  Please  do  not  hesitate  to 
contact  us  with  any  questions  you  may  have. 
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Establishing  Expert  Criterion 


Section  1:  Preliminary  Startup 

To  conduct  this  trail  successfully,  it  is  important  to  recruit  as  many  I^aroscopic  experts 
as  possible.  In  this  w^,  we  will  be  able  to  ensure  that  the  performance  criterion  levels 
that  are  set  for  the  residents  undergoing  curriculum  based  training  accurately  reflect  a 
relatively  normal  distribution  of  expertise  seen  throughout  participating  centers.  As  such, 
we  expect  a  number  of  experts  to  participate  from  each  center.  Currently  at  Emory, 
seven  experts  have  volunteered. 

Officially,  we  are  defining  an  “expert”  as  any  individual  that  has  performed  100  or  more 
laparoscopic  cholecystectomies.  Once  identified,  an  expert  will  perform  5  trials  of  the  6 
MIST-VR  Core  Skills  1  tasks  of  AcquirePlace,  TransferPIace,  Traversal,  Wilhdrawinsert, 
Diathermy,  and  Manipulation  Diathermy  using  die  specific  task  configurations  set  for  the 
MASTER  CBLC  multicenter  trial. 

Each  individual  trial  (of  all  6  tasks)  should  be  completed  in  its  entirety  during  a  given 
session.  Experts  may  complete  several  complete  trials  in  one  session  if  they  so  choose. 
We  recommend  that  they  not  complete  all  five  trials  in  a  single  session,  however.  On 
average,  each  trial  takes  approximately  forty-five  minutes  to  complete.  The  unique 
CBLC  task  settings  are  purposefully  difficult  and  will  take  longer  to  complete  than  when 
performed  on  the  default  easy,  medium,  or  even  hard,  settings  that  your  experts  m^  be 
accustomed  to. 
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If  an  expert  is  completely  unfamiliar  with  the  MIST-VR  environm^it,  a  practice  run  on 
the  medium  MIST-VR  task  settings  can  be  performed.  In  this  case,  a  detailed 
explanation  of  the  MIST-VR  scoring  matrix  (especially  the  errors  scored)  should  be 
undertakai  prior  to  the  first  performance  of  each  task,  unless  the  expert  is  familiar  with 
the  MIST-VR  environment.  If  you  are  unfamiliar  with  these  particular  metrics,  please 
refer  to  the  MIST-VR  documentation  provided  with  you  simulator. 

Each  expert  will  have  their  own  preference  as  to  how  they  would  like  their  environment 
set.  Often  darkened  lifting,  a  quiet  room,  or  adjustment  of  the  instruments  will  be 
requested.  To  accommodate  sudi  requests,  we  have  not  set  a  standard  testing 
environment.  Unusual  requests  (e.g.  sitting)  will  need  to  be  discussed  with  the  trial 
coordinators  prior  to  acceptance,  however. 

Lastly,  there  are  two  points  about  the  scoring  metrics  that  should  be  discussed  with 
experts  prior  to  starting  these  tasks.  First,  we  are  primarily  concerned  with  errors  and 
instrument  economy.  In  this  respect,  time  to  completion  is  weighed  less  heavily  in  our 
calculations.  Within  the  MIST-VR,  however,  time  is  a  significant  factor  affecting  the 
final  score.  For  this  reason,  final  scores  should  not  be  provided  as  a  gauge  for  expert 
performance  progression.  This  is  also  why  using  die  gr^hing  function  provided  by 
FrameSET  (e.g.  peer-to-peer  or  progression)  are  not  used  during  these  expert  trials. 
Second,  during  the  last  task  (manipulation  dialhenny)  it  is  okay  to  pull  the  sphere  out  of 
the  large  box  (after  initially  grasping  it)  in  order  to  position  the  diathermy  box  into  a 
safer,  more  acceptable  position.  The  MIST-VR  environment  will  sometimes  place  the 
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diathemiy  box  in  an  unsafe  position  (e.g.  completely  bdiind  the  sphere,  or  very  close  to 
the  grasping  instrum^t).  In  these  cases  we  prefer  to  score  safety  over  exceptional 
dexterity,  and  tiierefore  have  eliminated  this  particular  error  from  the  manipulation 
diaihem^  task.  The  precedent  behind  both  of  these  adjustments  was  set  during  previous 
trials  studying  the  effectiveness  of  the  MIST-VR  as  a  means  to  train  safer  surgeons. 
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Section  2:  Configuring  MASTER  CBLC  specific  settings 


Step  1;  Load  FrameSET  by  double  clicking  the  Frameset  icon  shown 
In  Figure  1.  If  you  cannot  find  this  icon  readily,  the  Frameset  program 
can  be  found  in  the  windows  folder:  (c:/Program  Files/FrameSET/0. 


Step  2;  Login  to  the  FrameSET  administrator  mode  by  using  your  administrative 
username  and  password.  If  you  do  not  remember  these  codes,  remember  that  Mentice 
occasionally  sets  the  administrator  default  to  username  =  a;  password  =  a 

Step  3:  Once  inside  the  FrameSET  ^minislrator  mode, 
left  click  on  the  plus  sign  to  the  left  of  the  FrameSET  menu 
choice  in  the  left  sided  window  pane  (Figure  2).  Several 
branches  will  present  under  the  FrameSET  data  tree.  In 
this  section  you  will  be  concerned  with  the  Tasks  option. 

Step  4:  Left  click  on  the  plus  sign  to  the  left  of  the  Tasks 
folder  as  shown  in  Figure  3.  The  branches  off  the  Tasks 
folder  are  all  the  MIST-VR  tasks  available  on  your 
computer.  Within  the  MASTER  CBLC  trial,  we  are  using 
the  Core  Skills  1  task  set.  This  includes  the  following 
tasks;  Acquire  Place,  Transfer  Place,  Traversal, 

Withdraw/Insert,  Diathermy,  and  Manipulation  Diathermy.  For  now,  you  will  set  a  new 
configuration  for  Acquire  Place. 
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Step  5;  Left  click  on  the  plus  sign  to  the  left  of  the  Acquire  Place  choice  (Figure  4). 

This  will  branch  into  Configs  and  Weightings.  Right  click  on  the  Configs  text  (not  on  the 
arrow),  and  choose  Add  Config  from  the  pop-up  menu  item.  In  the  right  window  pane,  a 
new  window  will  appear.  You  screen  should  look  like  Figure  4. 
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Step  6:  Change  the  name  of  the  New  Configuration  (circled  in  Figure  4),  to  MASTER 
CBLC.  Once  completed,  click  apply  (circled  in  Figure  5). 
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Step  7:  Direct  your  attention  to  the  circled  area  of 


Figure  6  (above).  Here  is  where  the  unique 


MASTER  CBLC  configuration  settings  will  be 


applied.  Change  these  settings  to  the  following 


specifications:  Target  Sphere  =  7.5;  Targ.  =  9.25 


(you  will  have  to  highlight  the  number  and  type  it  in 


yourself.  The  arrows  will  not  allow  you  to  go  below 
10);  Camera  Pos  =  100.  Once  complete,  click  on 
^ply,  and  check  to  make  sure  your  settings  match 


those  in  Figure  7.  When  complete,  close  the  configuration  window. 
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Step  8:  The  configuration  for  the  Acquire  Place  task  is  now  complete.  Take  similar 
steps  to  configure  the  remaining  5  tasks  using  the  settings  listed  in  Appendix  1,  Be  sure 
to  close  each  configuration  window  when  you  are  done  changing  the  individual  tasks  - 
otherwise  FrameSET  will  display  an  error  window. 
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Section  3:  Adding  Classes  and  Users  to  MIST-VR 


Step  1 :  Left  dick  the  plus  sign  to  the  left  of  the  Users  option  within  the  FrameSET  data 
tree  (Figure  1).  This  will  display  the  user  classes  (or  groups) 
currently  installed  within  your  copy  of  FrameSET.  In  this 
section  you  will  be  creating  your  Expert  class. 

Step  2;  The  easiest  way  to  create  a  class  is  to  left  dick  on  the 
Class  icon  located  on  the  FrameSET  menu  bar  at  the  top  of  the 
main  window  (Figure  2).  Next,  MASTER  CBLC  Expert 
for  Class  Name  at  the  prompt  (Figure  3). 
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When  you  are  finished  typing,  left  click  on  the  next  button  at  the  bottom  of  the  window. 
The  window  will  then  ask  whether  you  want  to  add  users  to  this  class.  We  have  found 
that  adding  users  in  this  fashion  will  crash  the  program;  therefore  you  do  not  want  to 
create  users  at  this  time.  Continue  on  by  clicking  the  Next  button  again.  Your  class 
should  appear  under  ihe  Users  data  tree  in  the  window  to  die  left. 
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Step  3:  To  add  users,  you  can  either  a)  left 
click  on  the  Users  icon  on  die  FrameSET 
menu  bar  (Figure  4);  or  b)  ri^t  click  on  die 
MASTER  Expert  Class  item  found  in  the  data 
tree  to  the  left. 

a)  If  you  use  the  Users  icon,  use  the  subject’s  number  as  the  account  name,  and  be  sure 

to  left  click  on  the  Enter  Now  radial  button  in 
order  to  add  more  details  (Figure  5).  Left  click 
the  Next  button  on  the  bottom  of  the  window  to 
continue.  Within  the  details  section,  use  the 
expert’s  number  as  his/her  last  name  (e.g.  Expert 
1).  Next,  choose  a  password  that  you  can 
remember.  Confirm  the  password  and  click  the 

Next  button.  The  next  window  will  ask  you  to  select  a  user  class  for  this  new  user.  For 
diis  example  you  will  choose  the  MASTER  Expert  class. 

b)  If  you  right  click  on  the  MASTER  Expert  Class  in 
the  data  tree,  add  the  subject’s  number  as  the  account 
name  and  user  first  name.  Next,  choose  a  password 
you  will  remember  (Figure  6).  Click  apply  when 
you  are  finished. 
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Section  4:  Creating  a  MIST-VR  task  setup  with  assignments 


Once  you  have  designed  a  class  and  placed  experts  into  it,  you  will  need  to  add  a  task 
setup  to  the  group.  The  task  setup  is  the  sequence  of  MIST-VR  tasks  that  each  user  will 
be  asked  to  perform.  Since  you  are  using  the  same  task  configuration  settings  dial  will  be 
used  when  residents  are  training,  you  can  use  the  same  setup  for  bodi  expert  and  training 
classes  (once  diat  class  is  made  later  in  the  trial).  Here  you  create  die  MASTER 
Training  setup  that  wall  be  used  for  both  of  these  groups  -  although  it  is  specifically  used 
for  the  experts  in  this  initial  portion  of  the  trial. 

Step  1;  Left  click  on  the  Setup  icon  located  on  the  FrameSET  menu  bar  at  the  top  of  die 

main  window  (Figure  1).  Type  A£45r£R  Training 
in  the  setup  name  box.  Move  on  to  the  next  step  b^' 
left  clicking  on  the  finish  button. 


Step  2:  You  will  need  to  add  content,  as 
introductory  segments  to  the  various  skill  tasks,  as 
w'ell  as  the  tasks  themselves.  Add  the  introduction 
to  the  acquire  place  MIST-VR  task  by  left  clicking 
on  the  add  button  under  the  components  portion  of 
the  window  (Figure  3).  When  the  pop-up  menu 
appears,  left  click  on  the  Content  optioa 
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Next,  left  click  the  arrow  to  the  left  of  the 
Expert  Demo  option.  Choose  the  AP  Demo 
option  and  dieck  to  make  sure  it  was  added  to 
your  setup  listing  (Figure  4). 


Step  3:  Now  add  die  Acquire  Place  task,  Left 
click  on  the  add  button  under  the  components 
portion  of  the  window.  When  the  pop-up 
appears,  choose  the  Task  optioa  Choose 
AcquirePIace  and  then  left  click  the  Nexi 
button.  A  new  window  displaying  the  various 
task  configuration  options  now  appears. 
Choose  the  Medium  configuration  and  left  click  on  the  Finish  button, 
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Step  4:  Continue  adding  content  and  MASTER 
CBLC  configuration  tasks  in  the  following  order. 
Content  ->  TP  demo;  Task  TransferPlace; 
Content  TV  demo;  Task  Trai'ersal;  Content 
->  W1  Insert;  Task  Withdrawlnsert;  Content 
DT  demo;  Task  Diathermy;  Content  MD 
demo;  Task  ^  Manipulation  Diathermy,  When 
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you  are  finished  your  window  should  look  like  Figure  5. 


Step  5:  Each  expert  will  need  to  be  assigned  a  task  setup 
before  using  the  MIST-VR  s>'stem.  Assipment  is 
accomplished  by  left  clicking  on  a  setup  and  dragging  the 
setup  to  a  given  user  name.  If  the  users  of  a  givai  class  are 
going  to  perform  tiie  same  tasks,  you  can  save  time  by 
dragging  the  setup  over  the  class  name  instead  of  each 
individual  user  (Figure  6).  For  the  MASTER  lap.  chole. 
trial  the  Training  setup  will  be  applied  to  the  MASTER 
Expert  class. 
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Section  5:  Exporting  MIST-Vn/FrameSET  data  to  Excel 


Three  sets  of  data  will  be  needed  for  completion  of  this  MASTER  trial:  the  expert 
performance  levels  for  each  task,  the  results  of  the  preliminary  psychomotor  testing  of  the 
individual  subjects,  and  the  performance  of  each  subject  during  training.  We  have 
chosen  to  accept  MIST-VR  data  for  analysis  in  the  Excel  spreadsheet  format. 
Unfortunately,  there  is  no  easy  way  to  export  all  MIST-VR  data  from  a  specific  user  class 
into  the  same  Excel  spreadsheet.  Instead,  FrameSET  only  allows  task  specific  data  to 
exist  on  the  same  spreadsheet.  For  example,  let’s  assume  you  have  two  experts  at  your 
institution  who  have  completed  their  5  MIST-VR  trials  to  help  set  the  expert  criterion  for 
the  study.  In  order  to  have  their  data  processed  for  statistical  analysis,  you  will  save  six 
different  Excel  files  -  one  for  each  individual  MIST-VR  task.  In  the  case  of  the 
AcquirePlace  task,  this  is  conceptually  depicted  in  Figxire  1. 
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In  the  same  example,  ifyou  enroll  eight  subjects  with  four  of  them  randomizing  to  VR 

training,  you  will  eventually 
send  a  total  of  36  Excel  files 
to  Emory.  This  is  shown  in 
Figure  2.  Also  note  thart  this 
figure  has  the  number  of 
rows  each  Excel  file  should 
contain  (e.g.  2  experts,  5 
trials  =  10  rows).  There  are 
question  marks  underneath 
the  tasks  performed  by  the 
trainees  due  to  our  inability 
to  predict  how  many  training  trials  it  will  take  to  reach  performance  criterion  levels. 

Now,  with  an  understanding  of  what  you  will  be  sending,  what  follows  is  a  description  of 
the  export  and  saving  process  using  the  psychomotor  testing  data  as  an  example.  Like 
previous  examples,  mimicking  these  steps  will  allow  you  to  save  and  package  the 
remaining  data  when  needed.  Ifyou  do  nothave  Excel 
installed  on  the  computers  running  FrameSET,  there  is  an 
alternative  work-around  that  is  described  in  Appendix  3. 

Step  1:  Navigate  through  the  Frameset  data  tree  (Irft 
clicking  the  plus  signs  to  the  left  of  the  text  items)  to  find  die 
users  in  the  MASTER  Expert  class.  Left  click  the  plus  sign 
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on  each  of  the  users  until  you  see  a  Results  item  under  each  user  (Figure  3). 

Step  2;  Left  dick  on  the  plus  sign  next  to  the  Results  item  under  die  first  user  listed  (a 
plus  sign  will  appear  after  the  expert  has  con^leted  a  trial).  The  MIST-VR  Core  Skills  1 
tasks  are  then  listed  beneath.  Left-click  the  plus  sign  next  to  the  AcquirePlace  task. 

Step  3:  Underneath  you  will  find  a  Cf  (or  configurations)  folder  named  Medium.  Within 

this  folder  are  the  raw  scores  of  the  user’s  one  trial  of  AcquirePlace  during  the  initial 
psychomotor  testing.  Left  click  on  the  plus  sign  to  see  the  Re  (or  Results)  folder  named 
with  the  date  and  time  the  task  was  performed. 

Step  4;  Right  click  on  the  results  folder  and  choose  the  Display  option  from  the  pop-up 
menu.  You  should  see  a  window  labeled  “Display  Results  -  AcquirePlace,  Medium 
Configuration”  in  the  right  sided  window  pane  (Figure  4). 


Step  5;  You  will  now  need  to  add  the  reminder  of  the  experts’  AcquirePlace  results  in 
the  display  results  window  by  repeating  Steps  2-4  for  each  user. 
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Step  6;  Export  the  data  to  an  Excel  spreadsheet  by  left  clicking  on  the  Export  button  on 
the  bottom  of  the  display  results  window  (Figure  5).  The  program  will  then  ask  for  a  file 
name.  As  listed  in  Appendix  2,  ihe  file  name  for  this  file  (collected  at  Emory)  should  be 
MastLCEmoryExpertAP.  Choose  a  folder  to  save  the  Excel  file  in,  and  left  dick  on  the 
Save  button  at  the  bottom  of  the  window. 
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Figure  5 


Step  7;  Repeat  Steps  2-6  for  eadi  respective  MIST-VR  task.  Use  the  naming 
conventions  listed  in  Appendix  2. 


Step  8:  Place  all  Excel  files  on  a  CD-Rom,  or  3.5”  floppy  disk  and  send  them  to  the 
Emory  Endosurgery  Center  as  listed  on  the  title  page. 
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Appendix  1:  MIST-VR  CBLC  configurations 


Target 

Sphere 

Target 

Box 

Diathermy 

Box 

Length 

Radius 

Number 

Gamer; . 
Positio  1 

Acquire  Place 

7.5 

9.25 

100 

Transfer  Place 

7.5 

9.25 

100 

Traversal 

1 

1 

1 

1 

1 

1 

1 

5 

2.5 

6 

100 

Withdraw/Insert 

7 

9 

100 

Diathermy 

4 

3.5 

100 

Manipulation 

7.5 

17.5 

6.5 

IOC 

Diathermy 

MASTER  CBLC  Guide 
Establishing  Expert  Criterion 


Appendix  2:  File  naming 


File  naming  template:  MastLC^TZ.xls 

MastLC  stands  for  MASTER  Laparoscopic  Cholecystectomy 
X  =  Institution  name 
Y  =  Stu(^  component 
Z  =  MIST-VRtask 


Institution  name  abbreviations 

Use  the  first  five  letters  of  your  institution  name.  Alternatively,  you  can  choose  an 
abbreviation  <  5  characters  that  mav  be  more  fitting  as  an  identifier  (e.g.  UNC). 

Studv  comnonent  abbreviations 

Setting  expert  oiterion 

Expert 

Psvdiomotor  testing 

Psvch 

Training 

Train 

MlST-VR  task  abbreviations 

AcauirePlace 

AP 

TransferPlace 

TP 

Traversal 

TV 

Withdrawinsert 

WI 

DT 

Manipulation  Diathermy 

MD 

Example:  For  a  group  of  Emoiy  University  expats  who  have  finished  their  5  MIST-VR 
trials,  the  name  for  the  Manipulation  Diathermy  task  results  file  should  be: 

MastLCEmor}'ExpertMD 
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Appendix  3:  Exporting  data  without  using  Excel 


This  appendix  is  intended  for  institutions  that  are  unable  to  use  Excel  as  their  means  of 
data  presentation  and  pack^ing  for  analysis.  To  see  how  to  manipulate  FrameSET  in 
order  to  export  user  data  for  the  MASTER  Laparoscopic  Cholecystectomy  trial,  please 
refer  to  Section  4  in  this  instruction  guide. 

To  export  your  data  you  will  need  to  have  a  text  editor  loaded  on  die  computer  the  data  is 
being  exported  from.  Examples  include  the  Microsoft  programs  NotePad,  or  WordPad 
that  are  provided  with  the  Microsoft  operating  system  software.  Most  likdy,  one  of  these 
programs  is  already  loaded  on  your  computer,  and  can  be  found  within  the  Accessories 
folder  located  within  the  programs  region  of  the  Start  menu. 

In  Section  4  of  the  instruction  guide,  Step  6  asks  you  to  click  on  an  export  button  that 
automatically  saves  data  in  an  Excel  format.  Instead  of  performing  this  task,  you  will 
need  to  cut  and  paste  all  of  the  user  data  from  the  results  window  into  your  text  editor. 

Step  1 :  Using  your  mouse,  highlight  all  the  users  and  their  results. 

Step  2;  Making  sure  that  all  the  data  remains  highlighted,  click  on  the  Copy  button  on 
the  lower  portion  of  the  window.  This  will  place  all  the  highli^ted  data  in  the  windows 
clipboard  so  that  you  can  paste  it  into  your  text  editing  program. 

Step  3:  Paste  the  data  into  your  text  editing  program  using  the  ^propriale  commands  for 
the  program.  In  NotePad  or  WordPad,  you  can  find  the  paste  command  under  the  Edit 
menu  item  on  the  top  toolbar. 

Step  4:  Save  the  data  using  the  appropriate  file  name  as  discussed  in  Appendix  2.  Files 
saved  using  this  method  will  have  a  .txt  extension  instead  of  the  .xls  extension  discussed 
above. 


MASTER  CBLC  Guide 
EsUbiishing  Expert  Criterion 


